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Abstract 

Nash equilibrium is the most commonly-used notion of equi- 
librium in game theory. However, it suffers from numerous 
problems. Some are well known in the game theory commu- 
nity; for example, the Nash equilibrium of repeated prisoner's 
dilemma is neither normatively nor descriptively reasonable. 
However, new problems arise when considering Nash equi- 
librium from a computer science perspective: for example, 
Nash equilibrium is not robust (it does not tolerate "faulty" 
or "unexpected" behavior), it does not deal with coalitions, it 
does not take computation cost into account, and it does not 
deal with cases where players are not aware of all aspects of 
the game. Solution concepts that try to address these short- 
comings of Nash equilibrium are discussed. 



1 Introduction 

Nash equilibrium is the most commonly-used notion of 
equilibrium in game theory. Intuitively, a Nash equilibrium 
is a strategy profile (a collection of strategies, one for each 
player in the game) such that no player can do better by 
deviating. The intuition behind Nash equilibrium is that it 
represent a possible steady state of play. It is a fixed point 
where each player holds correct beliefs about what other 
players are doing, and plays a best response to those beliefs. 
Part of what makes Nash equilibrium so attractive is that 
in games where each player has only finitely many possible 
deterministic strategies, and we allow mixed (i.e., random- 
ized) strategies, there is guaranteed to be a Nash equilibrium 
INas h 19501 (this was, in fact, the key result of Nash's the- 
sis). 

For quite a few games, thinking in terms of Nash equi- 
librium gives insight into what people do (there is a reason 
that game theory is taught in business schools!). However, 
as is well known, Nash equilibrium suffers from numerous 
problems. For example, the Nash equilibrium in games such 
as repeated prisoner's dilemma is to always defect (see Sec- 
tion[3]for more discussion of repeated prisoner's dilemma). 
It is hard to make a case that rational players "should" play 
the Nash equilibrium in this game when "irrational" players 
who cooperate for a while do much better! Moreover, in a 



game that is only played once, why should a Nash equilib- 
rium arise when there are multiple Nash equilibria? Players 
have no way of knowing which one will be played. And even 
in games where there is a unique Nash equilibrium (like re- 
peated prisoner's dilemma), how do players obtain correct 
beliefs about what other players are doing if the game is 
played only once? (See |Kreps 1990 1 for a discussion of 
some of these problems.) 

Not surprisingly, there has been a great deal of work 
in the economics community on developing alternative 
solution concepts. Various alternatives to and refine- 
ments of Nash equilibrium have been introduced, in- 
cluding, among many others, rationalizability , sequen- 
tial equilibrium, (trembling hand) perfect equilibrium, 
proper equilibrium, and iterated deletion of weakly dom- 
inated strategies. (These notions are discussed in stan- 



dard game theory text, such as |Fudenberg and Tirole 1991 
Osbo rne and Rubinstein 19941 .) Despite some successes 
none of these alternative solution concepts address the fol- 
lowing three problems with Nash equilibrium, all inspired 
by computer science concerns. 

• Although both computer science and distributed com- 
puting are concerned with multiple agents interacting, 
the focus in the game theory literature has been on the 
strategic concerns of agents — rational players choosing 
strategies that are best responses to strategies chosen 
by other player, the focus in distributed computing 
has been on problems such as fault tolerance and 
asynchrony, leading to, for example work on Byzan- 
tine agreement |Fischer, Lynch, and Paterson 1985| 
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Pease, Shostak, and Lamport 1980 1 . Nash equilibrium 
does not deal with "faulty" or "unexpected" behavior, nor 
does it deal with colluding agents. In large games, we 
should expect both. 

• Nash equilibrium does not take computational concerns 
into account. We need solution concepts that can deal 
with resource-bounded players, concerns that are at the 
heart of cryptography. 

• Nash equilibrium presumes that players have common 
knowledge of the structure of the game, including all the 
possible moves that can be made in every situation and all 
the players in game. This is not always reasonable in, for 
example, the large auctions played over the internet. 



In the following sections, I discuss each of these issues in 
more detail, and sketch solution concepts that can deal with 
them, with pointers to the relevant literature. 

2 Robust and Resilient Equilibrium 

Nash equilibrium tolerates deviations by one player. It is 
perfectly consistent with Nash equilibrium that two players 
could do much better by deviating in a coordinated way. For 
example, consider a game with n > 1 players where players 
much play either or 1 . If everyone plays 0, everyone get a 
payoff of 1 ; if exactly two players plays 1 and the rest play 
0, then the two who play 1 get a payoff of 2, and the rest get 
0; otherwise, everyone gets 0. Clearly everyone playing is 
a Nash equilibrium, but any pair of players can do better by 
deviating and playing 1 . 

Say that a Nash equilibrium is k-resilient if it tolerates 
deviations by coalitions of up to fc players. The notion of 
resilience is an old one in the game theory literature, going 
back to Aumann [1959|. Various extensions of Nash equi- 
librium have been proposed in the game theory literature to 



be viewed as a way of adding fault tolerance to equilib- 
rium notions. To capture this intuition, Abraham et al. 



| Abraham, Dolev, Gonen, and Halpern 2006 1 define a strat- 



deal with coalitions |Bernheim, Peleg, and Whinston 1989 
IMoreno and Wooders 1 99611 . However, these notions do not 
deal with players who act in unexpected ways. 

There can be many reasons that players act in unexpected 
ways. One, of course, is that they are indeed irrational. 
However, often seemingly irrational behavior can be ex- 
plained by players having unexpected utilities. For example, 
in a peer-to-peer network like Kazaa or Gnutella, it would 
seem that no rational agent should share files. Whether or 
not you can get a file depends only on whether other people 
share files. Moreover, there are disincentives for sharing (the 
possibility of lawsuits, use of bandwidth, etc.). Neverthe- 
less, people do share files. However, studies of the Gnutella 
network have shown that almost 70 percent of users share no 
files and nearly 50 percent of responses are from the top 1 
percent of sharing hosts lAd ar and Huberman 200011 . Is the 
behavior of the sharing hosts irrational? It is if we assume 
appropriate utilities. But perhaps sharing hosts get a big kick 
out of being the ones that provide everyone else with the mu- 
sic they play. Is that so irrational? In other cases, seemingly 
irrational behavior can be explained by faulty computers or 
a faulty network (this, of course, is the concern that work 
on Byzantine agreement is trying to address), or a lack of 
understanding of the game. 

To give just one example of a stylized game where this 
issue might be relevant, consider a group of n bargaining 
agents. If they all stay and bargain, then all get 2. However, 
if any agent leaves the bargaining table, those who leave get 
1, while those who stay get 0. Clearly everyone staying at 
the bargaining table is a fc-resilient Nash equilibrium for all 
k > 0, and it is Pareto optimal (everyone in fact gets the 
highest possible payoff). But, especially if n is large, this 
equilibrium is rather "fragile"; all it takes is one person to 
leave the bargaining table for those who stay to get 0. 

Whatever the reason, as pointed out by Abraham et 
al. 1120061 , it seems important to design strategies that tol- 
erate such unanticipated behaviors, so that the payoffs of 
the users with "standard" utilities do not get affected by 
the nonstandard players using different strategies. This can 



egy profile to be t-immune if no player who does not deviate 
is worse off if up to t players do deviate. Note the differ- 
ence between resilience and immunity. A strategy profile is 
resilient if deviators do not gain by deviating; a profile is 
immune if non-deviators do not get hurt by deviators. In the 
example above, although everyone bargaining is a fc-resilient 
Nash equilibrium for all k > 0, it is not 1 -immune. 

Of course, we may want to combine resilience and re- 
silience; a strategy is (fc, t)-robust if it is both fc-resilient and 
f-immune. (All the informal definitions here are completely 
formalized in (Abraham, Dolev, Gonen, and Halpern 2006 1 



Abraham, Dolev, and Halpern 2008J.) A Nash equilibrium 
is just a (l,0)-robust equilibrium. Unfortunately, for (fc, t) ^ 
(1,0), a (fc,£)-robust equilibrium does not exist in gen- 
eral. Nevertheless, there are a number of games of inter- 
est where they do exist; in particular, they can exist if play- 
ers can take advantage of a mediator, or trusted third party. 
To take just one example, consider Byzantine agreement 
[Pease, Shostak, and Lamport 1980|. Recall that in Byzan- 
tine agreement there are n soldiers, up to t of which may be 
faulty (the t stands for traitor), one of which is the general. 
The general has an initial preference to attack or retreat. We 
want a protocol that guarantees that (1) all nonfaulty soldiers 
reach the same decision, and (2) if the general is nonfaulty, 
then the decision is the general's preference. It is trivial 
to solve Byzantine agreement with a mediator: the general 
simply sends the mediator his preference, and the mediator 
sends it to all the soldiers. 

The obvious question of interest is whether we can 
implement the mediator. That is, can the players in 
the system, just talking among themselves (using what 
economists call "cheap talk") simulate the effects of the 
mediator. This is a question that has been of interest to 
both the computer science community and the game the- 
ory community. In game theory, the focus has been on 
whether a Nash equilibrium in a game with a mediator 
can be implemented using ch eap talk (cf. [Barany 1992} 
IBen-Porath 20031 |Forges 1990} IGerardi 2004 |Heller2005[ 
lUrbano and Vila 2002t lUrbano and Vila 20041 ). In cryptog- 
raphy, the focus has been on secure multiparty computation 
IIGoldreich et al. 1987t [Shamir etal. 19811 1 Yao 19821 . Here 
it is assumed that each agent i has some private information 
Xi (such private information, like the general's preference, 
is typically called the player's type in game theory). Fix a 
function /. The goal is have agent i learn f{x\, . . . , x n ) 
without learning anything about Xj for j ^ i beyond what 
is revealed by the value of f(xi, . . . , x n ). With a trusted 
mediator, this is trivial: each agent i just gives the mediator 
its private value xf, the mediator then sends each agent i the 
value f(xi, . . . , x n ). Work on multiparty computation pro- 
vides general conditions under which this can be done (see 
IIGold reich 2004] for an overview). Somewhat surprisingly, 
despite there being over 20 years of work on this problem 
in both computer science and game theory, until recently, 
there has been no interaction between the communities on 
this topic. 



Abraham et al. [2006 2008 1 essentially characterize when 
mediators can be implemented. To understand the results, 
three games need to be considered: an underlying game T, 
an extension Td of T with a mediator, and a cheap-talk exten- 
sion Tct of r. r is assumed to be a normal-form Bayesian 
game: each player has a type from some type space with a 
known distribution over types, and must choose an action 
(where the choice can depend on his type). The utilities of 
the players depend on the types and actions taken. For exam- 
ple, in Byzantine agreement, the possible types of the gen- 
eral are and 1, his possible initial preferences (the types of 
the other players are irrelevant). The players' actions are to 
attack or retreat. The assumption that there is a distribution 
over the general's preferences is standard in game theory, al- 
though not so much in distributed computing. Nonetheless, 
in many applications of Byzantine agreement, it seems rea- 
sonable to assume such a distribution. Roughly speaking, 
a cheap talk game implements a game with a mediator if it 
induces the same distribution over actions in the underlying 
game, for each type vector of the players. With this back- 
ground, I can summarize the results of Abraham et al. 

• If n > 3k + 3t, a (k, i)-robust strategy a with a medi- 
ator can be implemented using cheap talk (that is, there 
is a (k, i)-robust strategy a' in the cheap talk game such 
that a and a' induce the same distribution over actions 
in the underlying game). Moreover, the implementation 
requires no knowledge of other agents' utilities, and the 
cheap talk protocol has bounded running time that does 
not depend on the utilities. 

• If n < 3k + 3t then, in general, mediators cannot be im- 
plemented using cheap talk without knowledge of other 
agents' utilities. Moreover, even if other agents' utilities 
are known, mediators cannot, in general, be implemented 
without having a (k + i)-punishment strategy (that is, a 
strategy that, if used by all but at most k + t players, guar- 
antees that every player gets a worse outcome than they do 
with the equilibrium strategy) nor with bounded running 
time. 

• If n > 2k + 3t, then mediators can be implemented using 
cheap talk if there is a punishment strategy (and utilities 
are known) in finite expected running time that does not 
depend on the utilities. 

• If n < 2k + 3t then mediators cannot, in general, be im- 
plemented, even if there is a punishment strategy and util- 
ities are known. 

• If n > 2k + 2t and there are broadcast channels then, for 
all e, mediators can be e-implemented (intuitively, there 
is an implementation where players get utility within e of 
what they could get by deviating) using cheap talk, with 
bounded expected running time that does not depend on 
the utilities. 

• If n < 2k + 2t then mediators cannot, in general, be e- 
implemented, even with broadcast channels. Moreover, 
even assuming cryptography and polynomially-bounded 
players, the expected running time of an implementation 
depends on the utility functions of the players and e. 



• If n > k + 3t then, assuming cryptography and 
polynomially-bounded players, mediators can be e- 
implemented using cheap talk, but if n < 2k + 2t, then 
the running time depends on the utilities in the game and 
e. 

• If n < k + 3t, then even assuming cryptography, 
polynomially-bounded players, and a (k + t) -punishment 
strategy, mediators cannot, in general, be e-implemented 
using cheap talk. 

• If n > k + t then, assuming cryptography, polynomially- 
bounded players, and a public-key infrastructure (PKI), 
we can e-implement a mediator. 

All the possibility results showing that mediators can 
be implemented use techniques from secure multiparty 
computation. The results showing that that if n < 3k + 3t, 
then we cannot implement a mediator without knowing 
utilities and that, even if utilities are known, a punish- 
ment strategy is required, use the fact that Byzantine 
agreement cannot be reached if t < n/3; the impos- 
sibility result for n < 2k + 3t also uses a variant of 
Byzantine agreement. These results provide an excellent 
illustration of how the interaction between computer 
science and game theory can lead to fruitful insights. 
Related work on implementing mediators can be found 
in [Gordon and Katz 2006; Halpern and Teague 2004 
|Izmalkov, Micali, and Lepinski 2005| IKol and Naor 2008 



Lepinski, Micali, Peikert, and Shelat 2004 
Lysyanskaya and Triandopoulos 2006 1. 



3 Taking Computation Into Account 

Nash equilibrium does not take computation into account. 
To see why this might be a problem, consider the following 
example, taken from [Halpern and Pas s 2008) . 

Example 3.1: You are given a number n-bit number x. You 
can guess whether it is prime, or play safe and say nothing. 
If you guess right, you get $10; if you guess wrong, you lose 
$10; if you play safe, you get $1. There is only one Nash 
equilibrium in this 1 -player game: giving the right answer. 
But if n is large, this is almost certainly not what people will 
do. Even though primality testing can be done in polynomial 
time, the costs for doing so (buying a larger computer, for 
example, or writing an appropriate program), will probably 
not be worth it for most people. The point here is that Nash 
equilibrium is not taking the cost of computing whether x is 
prime into account. 

There have been attempts in the game theory community 
to define solution concepts that take computation into ac- 
count, going back to the work of Rubinstein |1986|. (See 
[ Kala i 19901 for an overview of the work in this area in the 
1980s, and ]Ben-Sasson, Kalai, and Ka lai 2007 1 for more 
recent work.) Rubinstein assumed that players choose a fi- 
nite automaton to play the game rather than choosing a strat- 
egy directly; a player's utility depends both on the move 
made by the automaton and the complexity of the automaton 
(identified with the number of states of the automaton). In- 
tuitively, automata that use more states are seen as represent- 
ing more complicated procedures. Rafael Pass and I [2008] 



provide a general game-theoretic framework that takes com- 
putation into account. (All the discussion in this section is 
taken from [Halpern and Pas s 2008] .) Like Rubinstein, we 
view all players as choosing a machine, but we use Turing 
machines, rather than finite automata. We associate a com- 
plexity, not just with a machine, but with the machine and its 
input. This is important in Example l3.ll where the complex- 
ity of computing whether x is prime depends, in general, on 
the length of x. 

The complexity could represent the running time of or 
space used by the machine on that input. The complexity 
can also be used to capture the complexity of the machine 
itself (e.g., the number of states, as in Rubinstein's case) or 
to model the cost of searching for a new strategy to replace 
one that the player already has. (One of the reasons that 
players follow a recommended strategy is that there may be 
too much effort involved in trying to find a new one; I return 
to this point later.) 

We again consider Bayesian games, where each player 
has a type. In a standard Bayesian game, an agent's util- 
ity depends on the type profile and the action profile (that is, 
every player's type, and the action chosen by each player). 
In a computational Bayesian game, each player i chooses 
a Turing machine. Player i's type t{ is taken to be the in- 
put to player i's Turing machine M,. The output of Mj on 
input ti is taken to be player i's action. There is also a com- 
plexity associated with the pair (M,-,£j). Player i's utility 
again depends on the type profile and the action profile, and 
also on the complexity profile. The reason we consider the 
whole complexity profile in determining player i's utility, as 
opposed to just i's complexity, is that, for example, i might 
be happy as long as his machine takes fewer steps than j's. 
Given these definitions, we can define Nash equilibrium as 
usual. With this definition, by defining the complexity ap- 
propriately, it will be the case that playing safe for suffi- 
ciently large inputs will be an equilibrium. 

Computational Nash equilibrium also gives a plausible 
explanation of observed behavior in finitely-repeated pris- 
oner's dilemma. 

Example 3.2: Recall that prisoner's dilemma, in prisoner's 
dilemma, there are two prisoners, who can choose to either 
cooperate or defect. As described in the table below, if they 
both cooperate, they both get 3; if they both defect, then both 
get 1 ; if one defects and the other cooperates, the defector 
gets 5 and the cooperator gets —5. (Intuitively, the cooper- 
ator stays silent, while the defector "rats out" his partner. If 
they both rat each other out, they both go to jail.) 
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(5,-5) 


(-3,-3) 



It is easy to see that defecting dominates cooperating: no 
matter what the other player does, a player is better off de- 
fecting than cooperating. Thus, "rational" players should 
defect. And, indeed, (D, D) is the only Nash equilibrium 
of this game. Although (C, C) gives both players a better 
payoff than (D, D), this is not an equilibrium. 



Now consider finitely repeated prisoner's dilemma 
(FRPD), where prisoner's dilemma is played for some fixed 
number N of rounds. The only Nash equilibrium is to al- 
ways defect; this can be seen by a backwards induction ar- 
gument. (The last round is like the one-shot game, so both 
players should defect; given that they are both defecting at 
the last round, they should both defect at the second-last 
round; and so on.) This seems quite unreasonable. And, 
indeed, in experiments, people do not always defect. In fact, 
quite often they cooperate throughout the game. Are they ir- 
rational? It is hard to call this irrational behavior, given that 
the "irrational" players do much better than supposedly ra- 
tional players who always defect. There have been many at- 
tempts to explain cooperation in FRPD in the literature (see, 
for example, (Kreps, Milgrom, Roberts, and Wilson 1982 1). 
Indeed, there have even been well-known attempts that take 
computation into account; it can be shown that if players 
are restricted to using a finite automaton with bounded com- 
plexity, then there exist equilibria that allow for cooper- 
ation [Ne yman 1985| [Pap adimitri ou and Yannakakis 1994) . 
However, the strategies used in those equilibria are quite 
complex, and require the use of large automata; as a conse- 
quence this approach does not seem to provide a satisfactory 
explanation as to why people choose to cooperate. 

Using the framework described above leads to a straight- 
forward explanation. Consider the tit-for-tat strategy, which 
proceeds as follows: a player cooperates at the first round, 
and then at round m + 1, does whatever his opponent did at 
round m. Thus, if the opponent cooperated at the previous 
round, then you reward him by continuing to cooperate; if he 
defected at the previous round, you punish him by defecting. 
If both players play tit-for-tat, then they cooperate through- 
out the game. Interestingly, tit-for-tat does exceedingly well 
in FRPD tournaments, where computer programs play each 
other OAxelrod 19841 . 

Tit-for-tat is a simple program, which needs very little 
memory. Suppose that we charge even a modest amount 
for memory usage, and that there is a discount factor S, with 
.5 < 6 < 1, so that if the player gets a reward of r m in round 
m, his total reward over the whole A^-round game is taken 

to be Ylm=i ^" lr m- I n tn ' s case > it is eas Y to see that, no 
matter what the cost of memory is, as long as it is positive, 
for a sufficiently long game, it will be a Nash equilibrium 
for both players to play tit-for-tat. For the best response to 
tit-for-tat is to play tit-for-tat up to the last round, and then 
to defect. But following this strategy requires the player to 
keep track of the round number, which requires the use of 
extra memory. The extra gain of $2 achieved by defecting at 
the last round, if sufficiently discounted, will not be worth 
the cost of keeping track of the round number. 

Note that even if only one player is computationally 
bounded and is charged for memory, and memory is free for 
the other player, then there is a Nash equilibrium where the 
bounded player plays tit-for-tat, while the other player plays 
the best response of cooperating up (but not including) to the 
round of the game, and then defecting. 

Although with standard games there is always a Nash 
equilibrium, this is not the case when we take computation 



into account, as the following example shows. 

Example 3.3c Consider roshambo (rock-paper-scissors). We 
model playing rock, paper, and scissors as playing 0, 1, and 
2, respectively. The payoff to player 1 of the outcome 
is 1 if i = j © 1 (where © denotes addition mod 3), —1 if 
j = i © 1, and if i = j. Player 2's playoffs are the negative 
of those of player 1; the game is a zero-sum game. As is 
well known, the unique Nash equilibrium of this game has 
the players randomizing uniformly between 0, 1, and 2. 

Now consider a computational version of roshambo. Sup- 
pose that we take the complexity of a deterministic strategy 
to be 1 , and the complexity of a strategy that uses random- 
ization to be 2, and take player i's utility to be his payoff in 
the underlying Bayesian game minus the complexity of his 
strategy. Intuitively, programs involving randomization are 
more complicated than those that do not randomize. With 
this utility function, it is easy to see that there is no Nash 
equilibrium. For suppose that (Mi, M2) is an equilibrium. 
If Mi uses randomization, then 1 can do better by playing 
the deterministic strategy j ffi 1, where j is the action that 
gets the highest probability according to M2 (or is the de- 
terministic choice of player 2 if M2 does not use random- 
ization). Similarly, M2 cannot use randomization. But it is 
well known (and easy to check) that there is no equilibrium 
for roshambo with deterministic strategies. 

Is the lack of Nash equilibrium a problem? Perhaps not. 
Taking computation into account should cause us to rethink 
things. In particular, we may want to consider other solution 
concepts. But, as the examples above show, Nash equilib- 
rium does seem to make reasonable predictions in a num- 
ber of games of interest. Perhaps of even more interest, us- 
ing computational Nash equilibrium lets us provide a game- 
theoretic account of security. 

The standard framework for multiparty security does not 
take into account whether players have an incentive to ex- 
ecute the protocol. That is, if there were a trusted media- 
tor, would player i actually use the recommended protocol 
even if i would be happy to use the services of the media- 
tor to compute the function /? Nor does it take into account 
whether the adversary has an incentive to undermine the pro- 
tocol. 

Roughly speaking, the game-theoretic definition says 
that II is a game-theoretically secure (cheap-talk) proto- 
col for computing / if, for all choices of the utility func- 
tion, if it is a Nash equilibrium to play with the me- 
diator to compute /, then it is also a Nash equilibrium 
to use II to compute /. Note that this definition does 
not mention privacy. It does not need to; this is taken 
care of by choosing the utilities appropriately. Pass and 
I 1 20081 show that, under minimal assumptions, this defi- 
nition is essentially equivalent to a variant of zero knowl- 
edge [Goldwasser, Micali, and Rackoff 1989| called precise 
zero knowledge [Micali and Pass 2006]. Thus, the two ap- 
proaches used for dealing with "deviating" players in two 
game theory and cryptography — Nash equilibrium and zero- 
knowledge "simulation " — are intimately connected; indeed, 
they are essentially equivalent once we take computation 
into account appropriately. 



4 Taking (Lack of) Awareness Into Account 

Standard game theory models implicitly assume that all sig- 
nificant aspects of the game (payoffs, moves available, etc.) 
are common knowledge among the players. However, this 
is not always a reasonable assumption. For example, sleazy 
companies assume that consumers are not aware that they 
can lodge complaints if there are problems; in a war setting, 
having technology that an enemy is unaware of (and thus be- 
ing able to make moves that the enemy is unaware of) can 
be critical; in financial markets, some investors may not be 
aware of certain investment strategies (complicated hedging 
strategies, for example, or tax-avoidance strategies). 

To understand the impact of adding the possibility of 
unawareness to the analysis of games, consider the game 
shown in Figure Q] (this example, and all the discussion in 
this section, is taken from [Halpern and Rego 2006]). One 
Nash equilibrium of this game has A playing across^ and 
B playing downs . However, suppose that A is not aware 
that B can play downs . In that case, if A is rational, A will 
play down^. Therefore, Nash equilibrium does not seem to 
be the appropriate solution concept here. Although A would 
play across^ if A knew that B were going to play down b, A 
cannot even contemplate this possibility, let alone know it. 



across^ 



down,! 



(1, 



B 



acrossj; 



down^ 



Figure 1 : A simple game. 

To find an appropriate analogue of Nash equilibrium in 
games where players may be unaware of some possible 
moves, we must first find an appropriate representation for 
such games. The first step in doing so is to explicitly repre- 
sent what players are aware of at each node. We do this by 
using what we call an augmented game. 

Recall that an extensive game is described by a game 
tree. Each node in the tree describes a partial history of the 
game — the sequence of moves that led to that node. Asso- 
ciated with each node is the player that moves at that node. 
Some nodes where a player i moves are grouped together 
into an information set for player i. Intuitively, if player 
i is at some node in an information set /, then i does not 
know which node of / describes the true situation; thus, at 
all nodes in /, i must make the same move. An augmented 
game is an extensive game with one more feature: associ- 
ated with each node in the game tree where player i moves 
is the level of awareness of player i — the set of histories that 
player i is aware of. 

We use the player's awareness level as a way of keep- 
ing track of how the player's awareness changes over time. 



For example, perhaps A playing across^ will result in B be- 
coming aware of the possibility of playing downs . In finan- 
cial settings, one effect of players using certain investment 
strategies is that other players become aware of the possibil- 
ity of using that strategy. Strategic thinking in such games 
must take this possibility into account. We would model 
this possibility by having some probability of B's awareness 
level changing. (The formal definition of augmented game 
can be found in |Halpern and Rego 2006 1.) 

For example, suppose that in the game shown in Figure Q] 

• players A and B are aware of all histories of the game; 

• player A is uncertain as to whether player B is aware of 
run (across^, downs) and believes that he is unaware of 
it with probability p; and 

• the type of player B that is aware of the run (across^, 
downs) is aware that player A is aware of all histories, 
and he knows A is uncertain about his awareness level 
and knows the probability p. 

Because A and B are actually aware of all histories of 
the underlying game, from the point of view of the modeler, 
the augmented game is essentially identical to the game de- 
scribed in Figure Q] with the awareness level of both players 
A and B consisting of all histories of the underlying game. 
However, when A moves at the node labeled A in the mod- 
eler's game, she believes that the actual augmented game is 
T A , as described in Figure [2] In T A , nature's initial move 
captures A's uncertainty about B's awareness level. At the 
information set labeled A A, A is aware of all the runs of 
the underlying game. Moreover, at this information set, A 
believes that the true game is T A . 

At the node labeled BA, B is aware of all the runs of 
the underlying game and believes that the true game is the 
modeler's game; but at the node labeled B.2, B is not aware 
that he can play downs, and so believes that the true game 
is the augmented game T B described in Figure [3] At the 
nodes labeled A. 3 and B.3 in the game T B , neither A nor B 
is aware of the move downs. Moreover, both players think 
the true game is T B . 
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Figure 2: The augmented game T . 

As this example should make clear, to model a game with 
possibly unaware players, we need to consider, not just one 
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Figure 3: The augmented game T 1 



augmented game, but a collection of them. Moreover, we 
need to describe, at each history in an augmented game, 
which augmented game the player playing at that history be- 
lieves is the actual augmented game being played. 

To capture these intuitions, starting with an underlying 
extensive-form game T, we define a game with awareness 
based on T to be a tuple T* = {G, T m , T), where 

• Q is a countable set of augmented games based on T, of 
which one is r m ; 

• T maps an augmented game T + 6 Q and a history h in 
T+ such that P+(h) = i to a pair (T h , I), where T h e Q 
and / is an information set for player i in game V . 

Intuitively, r™ is the game from the point of view of an om- 
niscient modeler. If player i moves at h in game T + 6 Q 
and J 7 (T + ,h) = {T h , I), then T h is the game that i believes 
to be the true game when the history is h, and / consists 
of the set of histories in r he currently considers possi- 
ble. For example, in the examples described in Figures [2] 
and [3] taking r™ to the augmented game in Figure Q] we 
have f(r ra , ( )) = (T A , I), where I is the information set 
labeled A A in Figure [2] and J 7 (T A , (unaware, across^)) = 
(T B , {(across^)}). There are a number of consistency con- 
ditions that have to be satisfied by the function T\ the details 
can be found in [Halpern and Rego 2006 1. 

The standard notion of Nash equilibrium consists of a pro- 
file of strategies, one for each player. Our generalization 
consists of a profile of strategies, one for each pair (i, T'), 
where T' is a game that agent i considers to be the true game 
in some situation. Intuitively, the strategy for a player i at V' 
is the strategy i would play in situations where i believes that 
the true game is V. To understand why we may need to con- 
sider different strategies consider, for example, the game of 
FigureQ] B would play differently depending on whether or 
not he was aware of downs- Roughly speaking, a profile a 
of strategies, one for each pair (i, T'), is a generalized Nash 
equilibrium if cr^r' is a best response for player i if the true 
game is V, given the strategi es cr 7i r" being used by th e other 
players in T'. As shown in [Halpern and Rego 2006], every 



game with awareness has a generalized Nash equilibrium. 

A standard extensive-form game T can be viewed as a 
special case of a game with awareness, by taking r m = T, 
Q = {r m }, and f(r, h) = (r m , I), where I is the infor- 
mation set that contains h. Intuitively, T corresponds to the 
game of awareness where it is common knowledge that T is 
being played. We call this the canonical representation of 
T as a game with awareness. It is not hard to show that a 



strategy profile a is a Nash equilibrium of T iff it is a gen- 
eralized Nash equilibrium of the canonical representation of 
r as a game with awareness. Thus, generalized Nash equi- 
librium can be viewed as a generalization of standard Nash 
equilibrium. 

Up to now, I have considered only games where players 
are not aware of their lack of awareness. But in some games, 
a player might be aware that there are moves that another 
player (or even she herself) might be able to make, although 
she is not aware of what they are. Such awareness of un- 
awareness can be quite relevant in practice. For example, in 
a war setting, even if one side cannot conceive of a new tech- 
nology available to the enemy, they might believe that there 
is some move available to the enemy without understanding 
what that particular move is. This, in turn, may encourage 
peace overtures. To take another example, an agent might 
delay making a decision because she considers it possible 
that she might learn about more possible moves, even if she 
is not aware of what these moves are. 

Although, economists usually interpret awareness as "be- 
ing able to conceive about an event or a proposition", there 
are other possible meanings for this concept. For exam- 
ple, awareness may also be interpreted as "understanding 
the primitive concepts in an event or proposition", or as 
"being able to determine if an event occurred or not", or 
as "being able to compute the consequences of some fact" 
[Fagin and Halpern 1988]. If we interpret "lack of aware- 
ness" as "unable to compute" (note that this interpretation 
is closely related to the discussion of the previous section!), 
then awareness of unawareness becomes even more signif- 
icant. Consider a chess game. Although all players under- 
stand in principle all the moves that can be made, they are 
certainly not aware of all consequences of all moves. A more 
accurate representation of chess would model this computa- 
tional unawareness explicitly. We provide such a represen- 
tation. 

Roughly speaking, we capture the fact that player i is 
aware that, at a node h in the game tree, there is a move 
that j can make she (i) is not aware by having i's subjective 
representation of the game include a "virtual" move for j at 
node h. Since i might have only an incomplete understand- 
ing of what can happen after this move, i simply describes 
what she believes will be the game after the virtual move, to 
the extent that she can. In particular, if she has no idea what 
will happen after the virtual move, then she can describe her 
beliefs regarding the payoffs of the game. Thus, our rep- 
resentation can be viewed as a generalization of how chess 
programs analyze chess games. They explore the game tree 
up to a certain point, and then evaluate the board position at 
that point. We can think of the payoffs following a virtual 
move by j in z's subjective representation of a chess game 
as describing the evaluation of the board from z's point of 
view. This seems like a much more reasonable representa- 
tion of the game than the standard complete game tree! 

All the definitions of games with awareness can be gener- 
alized to accommodate awareness of unawareness. In partic- 
ular, we can define a generalized Nash equilibrium as before, 
and once again show that every game with awareness (in- 
cluding awareness of unawareness) has a generalized Nash 



equilibrium | Halpern and Rego 2006| . 

There has been a great deal of work recently on 
modeling unawareness in games. The first papers on 
the topic w as by Feinberg M2004I 120051 . My work 
with Rego [2006 1 was the first to consider awareness 
in extensive games, modeling how awareness changed 
over time. There has been a recent flurry on the 
topic in the economics literature; see, for example, 
jHeifetz, Meier, and Schipper 2006b[ ILi 2006al ILi 2006bl 
Ozba y 2007[ . Closely related is work on logics that include 
awareness. This work started in the computer science 
literature | Fagin and Halpern 1988 1, but more recently, the 
bulk of the work has appeared in the economics literature 
(see, for example, |Dekel, Lipman, and Rustichini 1998 
|Halpern 200lj |Halpern and Rego 2008 



|Heifetz, Meier, and Schipper 2006a 
iModica and Rustichini 1994t 
IModic a and Rustich inTl999ll ). 



5 Conclusions 

I have considered three ways of going beyond standard Nash 
equilibrium, which take fault tolerance, computation, and 
lack of awareness into account, respectively. These are 
clearly only first steps. Here are some directions for fur- 
ther research (some of which I am currently engaged in with 
my collaborators): 

• For example, while (k, i)-robust equilibrium does seem to 
be a reasonable way of capturing some aspects of robust- 
ness, for some applications, it does not go far enough. I 
said earlier that in economics, all players were assumed 
to be strategic, or "rational"; in distributed computing, 
all players were either "good" (and followed the recom- 
mended protocol) or "bad" (in which case they could be 
arbitrarily malicious). Immunity takes into account the 
bad players. The definition of immunity requires that the 
rational players are not hurt no matter what the "bad" 
players do. But this may be too strong. As Ayer et 
al. 11200511 point out, it is reasonable to expect a certain 
fraction of players in a system to be "good" and follow 
the recommended protocol, even if it is not a best reply. 
In general, it may be hard to figure out what the best reply 
is, so if following the recommended protocol is not unrea- 
sonable, they will do that. (Note that this can be captured 
in a computational model of equilibrium, by charging for 
switching from the recommended strategy.) 

There may be other standard ways that players act irra- 
tional. For example, Kash, Friedman, and I [2003 con- 
sider scrip systems, where players perform work in ex- 
change for scrip. There is a Nash equilibrium where ev- 
eryone uses a threshold strategy, performing work only 
when they have less scrip than some threshold amount. 
Two standard ways of acting "irrationally" in such a sys- 
tem are to (a) hoard scrip and (b) provide service for free 
(this is the analogue of posting music on Kazaa). A ro- 
bust solution should take into account these more standard 
types of irrational behavior, without perhaps worrying as 
much about arbitrary irrational behavior. 



• The definitions of computational Nash equilibrium con- 
sidered only Bayesian games. What would appropriate 
solution concepts be for extensive-form games? Some 
ideas from the work on awareness seem relevant here, es- 
pecially if we think of "lack of awareness" as "unable to 
compute". 

• Where do the beliefs come from in an equilibrium with 
awareness? That is, if I suddenly become aware that you 
can make a certain move, what probability should I assign 
to you making that move? Ozbay [2007| proposes a so- 
lution concept where the beliefs are part of the solution 
concept. He considers only a simple setting, where one 
player is aware of everything (so that revealing informa- 
tion is purely strategic). Can his ideas be extended to a 
more general setting? 

Agents playing a game can be viewed participat- 
ing in a concurrent, distributed protocol. Game the- 
ory does not take the asynchrony into account, but 
it can make a big difference. For example, all the 
results from | Abraham, Dolev, Gonen, and Halpern 2006; 



|Abraham, Dolev, and Halpern 2008 1 mentioned in Section|2j 
depend on the system being synchronous. Things are more 
complicted in asynchronous settings. Getting solution con- 
cepts and that deal well with with asynchrony is clearly im- 
portant. 

Another issue that plays a major role in computer science 
but has thus far not been viewed as significant in game the- 
ory, but will, I believe, turn out to be important to the prob- 
lem of defining appropriate solution concepts, is the ana- 
logue of specifying and verifying programs. Games are typ- 
ically designed to solve certain problems. Thus, for exam- 
ple, economists want to design a spectrum auction so that 
the equilibrium has certain features. As I pointed out in an 
earlier overview [ Halper n" 2003| , game theory has typically 
focused on "small" games: games that are easy to describe, 
such as Prisoner's Dilemma. The focus has been on sub- 
tleties regarding basic issues such as rationality and coor- 
dination. To the extent that game theory is used to tackle 
larger, more practical problems, and especially to the ex- 
tent that it is computers, or software agents, playing games, 
rather than people, it will be important to specify carefully 
exactly what a solution to the game must accomplish. For 
example, in the context of a spectrum auction, a specifica- 
tion will have to address what should happen if a computer 
crashes while an agent is in the middle of transmitting a bid, 
how to deal with agents bidding on slow lines, dealing with 
agents who win but then go bankrupt, and so on. 

Finding logics to reason about solutions, especially do- 
ing so in a way that takes into account robustness and asyn- 
chrony, seems to me a difficult and worthwhile challenge. 
Indeed, one desideratum for a good solution concept is that 
it should be easy to reason about. Pursuing this theme, com- 
puter scientists have learned that one good way of designing 
correct programs is to do so in a modular way. Can a similar 
idea be applied in game theory? That is, can games designed 
for solving smaller problems be combined in a seamless way 
to solve a larger problem. If so, results about composability 
of solutions will be needed; we might want a solution con- 



cept that allows for such composability. 
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